The present disclosure is related to a method and system for presenting at least part of an image of a real object in a view of a real environment, and to method and system for selecting a subset of a plurality of images and determining at least part of a real object in at least one of the subset of the plurality of images for mapping to a view of a real environment.
Applications are known which augment a view of a real environment with one or more virtual objects using the so-called augmented reality (AR) technology. The view of a real environment may be captured as an image by a camera or captured as a visual impression by an eye, e.g. by using a so-called optical see-through device.
It is desired to correctly augment the captured view of the real environment with any virtual objects such that the virtual objects are merged with the view in a manner that the virtual objects fit in a perspectively and dimensionally correct fashion into the view of the real environment. For this, there is the need to determine the position and orientation (so-called pose) of a camera with respect to at least one part of the real environment in order to correctly augment the view of the real environment with any virtual objects.
The virtual object may be a computer generated object with a known 3D geometrical model (e.g. a CAD model or a set of 3D points). In this case, having a camera pose, the virtual object could be directly mapped or projected to a target area in an image of the real environment in a perspectively and dimensionally correct fashion. The virtual object may also be from at least part of an image of a real object captured by a camera. The image of the real object may be called a first image, and the image of the real environment may be called a second image, as also used herein below.
It is quite challenging to merge a part of the first image of the real object with the second image of the real environment in a perspectively and dimensionally correct fashion. The difficulty or challenge comes from the fact that the first image captures the real object from only one perspective view, wherein it may be difficult, or hardly possible, to represent this real object in another perspective view (e.g. the second image). Further, determining two camera poses in two different real environments where the first and second images are respectively captured is also quite difficult and prone to errors.
However, many AR applications exist, like furniture planning and room decoration, which would benefit from augmenting a view of the real environment (e.g. the second image of the real environment) with at least part of a first image of the real object. For example, the user could capture a camera image of a chair in a furniture store and augment his or her living room by overlaying the image of the chair onto a camera image of the living room. Then, the user may view the overlay on, for example, a display screen.
US 2010/0287511 A1 discloses methods and systems to merge a virtual object with a 2D image of a real environment or, by means of an optical, semi-transparent element, directly with the real environment. The virtual object is a computer generated 3D model. The method solves the problem of mutual concealments of the real environment and of the virtual object according to depth information and texture information associated with the 2D image.
US 2010/0208057 A1 discloses methods and systems for determining the pose of a camera with respect to at least one object of a real environment. It is proposed to manipulate a virtual reference model (i.e. a computer generated 3D model) in a 2D image of a real environment captured by a camera and determine distance data from a position of the virtual reference model resulted from the manipulation.
It would be beneficial to provide a method and system which supports merging or overlaying at least part of a real object of a first image with a second image of a real environment in a correct or an approximate correct perspective fashion.